Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix closing asynchronous handler on Python 3.12. #207

Closed
wants to merge 1 commit into from
Closed

Fix closing asynchronous handler on Python 3.12. #207

wants to merge 1 commit into from

Conversation

jgoclawski
Copy link

Consider the following example code:

import logging
from fluent import asynchandler

logging.basicConfig(level=logging.INFO)
l = logging.getLogger('fluent.test')
h = asynchandler.FluentHandler('app.follow', host='host', port=24224)
l.addHandler(h)
# l.info({
#   'from': 'userA',
#   'to': 'userB'
# })

On Python 3.12 this throws the following error:

Exception ignored in atexit callback: <function shutdown at 0x7ef7a54709a0>
Traceback (most recent call last):
  File "/home/x/.pyenv/versions/3.12.2/lib/python3.12/logging/__init__.py", line 2272, in shutdown
    h.close()
  File "/home/x/dev/fluent-logger-python/fluent/asynchandler.py", line 24, in close
    super().close()
  File "/home/x/dev/fluent-logger-python/fluent/handler.py", line 267, in close
    self.sender.close()
    ^^^^^^^^^^^
  File "/home/x/dev/fluent-logger-python/fluent/handler.py", line 214, in sender
    self._sender = self.getSenderInstance(
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/x/dev/fluent-logger-python/fluent/handler.py", line 240, in getSenderInstance
    return sender_class(
           ^^^^^^^^^^^^^
  File "/home/x/dev/fluent-logger-python/fluent/asyncsender.py", line 85, in __init__
    self._send_thread.start()
  File "/home/x/.pyenv/versions/3.12.2/lib/python3.12/threading.py", line 992, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't create new thread at interpreter shutdown

On Python 3.12 starting a new thread during interpreter shutdown raises an exception. When using fluent-logger-python such exception can happen when a process configured fluent handler, but did not use it to emit any records, so there was no sender created and no thread active.
At the same time - handler is closed automatically with atexit hook (that's a good thing) by default, so on interpreter shutdown. Calling handler.close() on a handler that did not emit any logs before, makes it create a sender, so that it can be immediately closed as well. But creating a sender also tries to create a thread - and this fails.

This commit takes the following approach: use the default asyncsender when possible, but if it throws RuntimeError it means that the interpreter is shutting down - so let's use the synchronous sender instead, so that the main thread is used instead of spawning a new one.

BTW this is a common issue affecting every Python library that uses threads to send messages. Here's a similar issue for Sentry: getsentry/sentry-python#2299
and a fix: getsentry/sentry-python#2468

@methane
Copy link
Member

methane commented Jun 5, 2024

Would you check #208? I think it is simpler solution.

@jgoclawski jgoclawski mentioned this pull request Jun 6, 2024
Merged
@jgoclawski
Copy link
Author

Thanks for taking the time to look into this! Closing in favour of #208.

@jgoclawski jgoclawski closed this Jun 6, 2024
@jgoclawski jgoclawski deleted the python_3.12 branch June 6, 2024 15:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jgoclawski @methane